Automatically computing emotions aroused from images through shape modeling

ABSTRACT

Shape features in natural images influence emotions aroused in human beings. An in-depth statistical analysis helps to understand the relationship between shapes and emotions. Through experimental results on the International Affective Picture System (IAPS) dataset, evidence is presented as to the significance of roundness-angularity and simplicity-complexity on predicting emotional content in images. Shape features are combined with other state-of-the-art features to show a gain in prediction and classification accuracy. Emotions are modeled from a dimensional perspective in order to predict valence and arousal ratings, which have advantages over modeling the traditional discrete emotional categories. Images are distinguished vis-a-vis strong emotional content from emotionally neutral images with high accuracy. All of the methods and steps disclosed herein are implemented on a programmed digital computer, which may be a stand-alone machine or integrated into another piece of equipment such as a digital still or video camera including, in all embodiments, portable devices such as smart phones.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/809,499, filed Jul. 27, 2015, which is a continuation-in-part of U.S.patent application Ser. No. 13/963,039, filed Aug. 9, 2013, nowabandoned, which claims priority from U.S. Provisional PatentApplication Ser. No. 61/683,845, filed Aug. 16, 2012, the entire contentof all of which is incorporated herein by reference.

GOVERNMENT SUPPORT

This invention was made with government support under Contract Nos.1110970 and 0821527 awarded by National Science Foundation. TheGovernment has certain rights in the invention.

FIELD OF THE INVENTION

This invention relates generally to computer-based image processing and,in particular, to automatically modeling the emotional content of animage from a dimensional perspective to predict valence and arousalratings.

BACKGROUND OF THE INVENTION

The study of human visual preferences and the emotions imparted byvarious works of art and natural images has long been an active topic ofresearch in the field of visual arts and psychology. A computationalperspective to this problem has interested many researchers and resultedin articles on modeling the emotional and aesthetic content in images[10, 11, 13]. However, there is a wide gap between what humans canperceive and feel and what can be explained using current computationalimage features. Bridging this gap is considered the “holy grail” ofcomputer vision and the multimedia community.

There have been many psychological theories suggesting a link betweenhuman affective responses and the low-level features in images apartfrom the semantic content. For example, studies indicate that roundnessand complexity of shapes are fundamental to understanding emotions.Studies of roundness indicate that geometric properties of visualdisplays convey emotions like anger and happiness. Bar et al. [5]confirm the hypothesis that curved contours lead to positive feelingsand that sharp transitions in contours trigger a negative bias. Withrespect to the complexity of shapes, and as enumerated in various worksof art, humans visually prefer simplicity. Any stimulus pattern isalways perceived in the most simplistic structural setting. Though theperception of simplicity is partially subjective to individualexperiences, it can also be highly affected by two objective factors,parsimony and orderliness. Parsimony refers to the minimalisticstructures that are used in a given representation, whereas orderlinessrefers to the simplest way of organizing these structures [3].

These findings provide an intuitive understanding of the low-level imagefeatures that motivate the affective response, but the small scale ofstudies from which the inferences have been drawn makes the results lessconvincing. In order to make a fair comparison of observations,psychologists created the standard International Affective PictureSystem (IAPS) [15] dataset by obtaining user ratings on three basicdimensions of affect, namely valence, arousal, and dominance (FIG. 1).However, the computational work on the IAPS dataset to understand thevisual factors that affect emotions has been preliminary. Researchers[9, 11, 18, 23, 25, 26] investigated factors such as color, texture,composition, and simple semantics to understand emotions, but have notquantitatively addressed the effect of perceptual shapes.

Previous work [11, 26, 18] predicted emotions aroused by images mainlythrough training classifiers on visual features to distinguishcategorical emotions, such as happiness, anger, and sad. Low-levelstimuli such as color and composition have been widely used incomputational modeling of emotions. Affective concepts were modeledusing color palettes, which showed that the bag of colors and Fishervectors (i.e., higher order statistics about the distribution of localdescriptors) were effective [9].

The study that did explore shapes by Zhang et al. [27] predictedemotions evoked by viewing abstract art images through low-levelfeatures like color, shape, and texture. However, this work only handlesabstract images, and focused on the representation of textures withlittle accountability of shape. Zhang et al. characterized shape throughZernike features, edge statistics features, object statistics, and Gaborfilters.

Emotion-histogram and bag-of-emotion features were used to classifyemotions by Solli et al. [24]. These emotion metrics were extractedbased on the findings from psycho-physiological experiments indicatingthat emotions can be represented through homogeneous emotion regions andtransitions among them.

The first work that comprehensively modeled categorical emotions,Machajdik and Hanbury [18] used color, texture, composition, content,and semantic level features such as number of faces to model eightdiscrete emotional categories. Besides the eight basic emotions, tomodel categorized emotions, adjectives or word pairs were used torepresent human emotions. The earliest work based on the Kansei systememploys 23 word pairs (e.g., like-dislike, warm-cool, cheerful-gloomy)to establish the emotional space [23]. Along the same lines, researchersenumerated more word pairs to reach a universal, distinctive, andcomprehensive representation of emotions in Wang et al. [25]. Yet, theaforementioned approaches of emotion representation ignore theinterrelationship among types of emotions.

SUMMARY OF THE INVENTION

This invention represents an attempt to systematically investigate howperceptual shapes contribute to emotions aroused from images throughmodeling the visual properties of roundness, angularity and simplicityusing shapes. Unlike edges or boundaries, shapes are influenced by thecontext and the surrounding shapes influence the perception of anyindividual shape [3]. To model these shapes in the images, the disclosedframework statistically analyzes the line segments and curves extractedfrom strong continuous contours. Investigating the quantitativerelationship between perceptual shapes and emotions aroused from imagesis non-trivial. First, emotions aroused by images are subjective. Thus,individuals may not have the same response to a given image, making therepresentation of shapes in complex images highly challenging. Second,images are not composed of simple and regular shapes, making itdifficult to model the complexity existing in natural images [3].

Leveraging the proposed shape features, the method seeks toautomatically distinguish the images with strong emotional content fromemotionally neutral images. In psychology, emotionally neutral imagesrefer to images which evoke very weak or no emotions in humans.

The approach models emotions from a non-\break categorical or discreteemotional perspective. In previous work, emotions were distinctlyclassified into categories like anger, fear, disgust, amusement, awe,and contentment\break among others. This invention represents the firstto predict emotions aroused from images by adopting a dimensionalrepresentation (FIG. 2). Valence represents the positive or negativeaspect of human emotions, where common emotions, like joy and happiness,are positive, whereas anger and fear are negative. Arousal describes thehuman physiological state of being reactive to stimuli. A higher valueof arousal indicates higher excitation. Dominance represents thecontrolling nature of the emotion. For instance, anger can be morecontrolling than fear. Researchers [2, 12, 28] have investigated theemotional content of videos through the dimensional approach. Theiremphasis was on the accommodation of the change in features over timerather than low-level feature improvement. However, static images, withless information, are often more challenging to interpret. Low-levelfeatures need to be punctuated.

This invention adopts the dimensional approaches of emotion motivated byrecent studies in psychology, which argued for the strengths ofdimensional approaches. According to Bradley and Lang [6], categorizedemotions do not provide a one-to-one relationship between the contentand emotion of an image since participants perceive different emotionsin the same image. This highlights the utility of a dimensionalapproach, which controls for the intercorrelated nature of humanemotions aroused by images. From the perspective of neurosciencestudies, it has been demonstrated that the dimensional approach is moreconsistent with how the brain is organized to process emotions at theirmost basic level [14, 17]. Dimensional approaches also allow theseparation of images with strong emotional content from images with weakemotional content.

Points of novelty of the invention include the ability to systematicallyinvestigate the correlation between visual shapes and emotions arousedfrom images. The concepts of roundness-angularity andsimplicity-complexity are quantitatively modeled from the perspective ofshapes using a dimensional approach, and images with strong emotionalcontent are distinguished from those with weak emotional content.Importantly, the method can automatically compute the valence and/orarousal coordinates of an image in the dimensional space model ofemotions.

Building upon the shape features, we have also investigated three visualcharacteristics of complex scenes that evoked human emotion utilizing alarge collection of ecologically valid image stimuli. Three newconstructs were developed that mapped the visual content to the scalesof roundness, angularity, and simplicity. Results of correlationalanalyses, between each construct and each dimension of emotionalresponses, showed that some of the correlations are statisticallysignificant, e.g., simplicity and valence, angularity and valence. Andclassification results demonstrated the capacity of the three constructsin classifying both dimensions of emotion. Interestingly, by combiningwith color features, the three constructs showed comparableclassification accuracy on distinguishing positive emotions fromnegative ones as a set of 200 texture, composition, facial, and shapefeatures. The invention may contribute to research regarding visualcharacteristics of complex scenes and human emotion from perspectives ofvisual arts, psychology, and computer science.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows example images derived from IAPS (The InternationalAffective Picture System) dataset. Images with positive affect from leftto right, and high arousal from bottom to top;

FIG. 2 is a dimensional representation of emotions and the location ofcategorical emotions in these dimensions (Valance, Arousal, andDominance);

FIG. 3 shows perceptual shapes of images with high valance;

FIG. 4 shows perceptual shapes of images with low valance;

FIG. 5 shows perceptual shapes of images with high arousal;

FIG. 6 shows perceptual shapes of images with low arousal;

FIG. 7A depicts a corner point;

FIG. 7B shows a point of inflexion;

FIG. 8 provides images with low mean value of the length of linesegments and their associated orientation histograms. The first row isthe original images; the second row shows the line segments; and thethird row shows the 18-bin histogram for line segments in the images;

FIG. 9 provides images with high mean value of the length of linesegments and their associated orientation histograms. The first row isthe original images; the second row shows the line segments; and thethird row shows the 18-bin histogram for line segments in the images;

FIG. 10 presents images with highest and lowest number of angles;

FIG. 11 illustrates the distribution of angles in images;

FIG. 12 shows images having a highest degree of curving;

FIG. 13 shows images having a lowest degree of curving;

FIG. 14A plots the valence distribution of ratings in IAPS;

FIG. 14B plots the arousal distribution of ratings in IAPS;

FIGS. 15A and 15B present charts showing the classification accuracy foremotional images and neutral images;

FIG. 16 are examples of misclassification in Set 1. The four rows areoriginal images, image contours, line segments, and continuous lines;

FIG. 17 are examples of misclassification in Set 2. The four rows areoriginal images, image contours, line segments, and continuous lines;

FIG. 18A presents experimental results as Mean squared error for thedimensions of valance and arousal;

FIG. 18B shows accuracy for the classification task;

FIG. 19 provides examples of images in the EmoSet;

FIGS. 20A-20D show mean value distributions of valence, arousal,dominance, and likeliness in the EmoSet;

FIG. 21 gives examples of images and their scores of roundness;

FIG. 22 provides examples of images and their scores of angularity;

FIG. 23 illustrates examples of images and their scores of simplicity;

FIGS. 24A-24D show the correlation between roundness and valence,arousal, dominance, and likeliness in natural photographs;

FIGS. 25A-25D depict the correlation between angularity and valence,arousal, dominance, and likeliness in natural photographs; and

FIGS. 26A-26D show the correlation between simplicity and valence,arousal, dominance, and likeliness in natural photographs.

DETAILED DESCRIPTION OF THE INVENTION

In accordance with this invention, emotions evoked by images arecaptured by leveraging shape descriptors. Shapes in images are difficultto capture, mainly due to the perceptual and merging boundaries ofobjects which are often not easy to differentiate using evenstate-of-the-art segmentation or contour extraction algorithms. Incontemporary computer vision literature [7, 20], there are a number ofstatistical representations of shape through characteristics like thestraightness, sinuosity, linearity, circularity, elongation,orientation, symmetry, and the mass of a curve. We choseroundness-angularity and simplicity-complexity characteristics becausethey have been found previously by psychologists to influence the affectof human beings through controlled human subject studies. Symmetry isalso known to effect emotion and aesthetics of images [22]. However,quantifying symmetry in natural images is challenging.

To make it more convenient to introduce the shape features proposed, thefour terms used are defined as: line segments, angles, continuous lines,and curves. The framework for extracting perceptual shapes through linesand curves is derived from [8]. The contours are extracted using thealgorithm in [1], which used color, texture, and brightness of eachimage for contour extraction. The extracted contours are of differentintensities and indicate the algorithm's confidence on the presence ofedges. Considering the temporal resolution of our vision system, weadopted a threshold of 40%. Example results are presented in FIGS. 3, 4,5, and 6. Pixels with an intensity higher than 40% are treated equally,which results in the binary contour map presented in the second column.The last three columns show the line segments, continuous lines, andcurves.

Line segments—Line segments refer to short straight lines generated byfitting nearby pixels. We generated line segments from each image tocapture its structure. From the structure of the image, we propose tointerpret the simplicity-complexity. We extracted locally optimized linesegments by connecting neighboring pixels from the contours extractedfrom the image [16].

Angles—Angles in the image are obtained by calculating angles betweeneach of any two intersecting line segments extracted previously.According to Julian Hochberg's theory [3], the number of angles and thenumber of different angles in an image can be effectively used todescribe its simplicity-complexity. The distribution of angles alsoindicates the degree of angularity of the image. A high number of acuteangles make an image more angular.

Continuous lines—Continuous lines are generated by connectingintersecting line segments having the same orientations with a smallmargin of error. Line segments of inconsistent orientations can becategorized as either corner points or points of inflexion. Cornerpoints, shown in FIG. 7A, refer to angles that are lower than 90degrees. Inflexion points, shown in FIG. 7B, refer to the midpoint oftwo angles with opposite orientations. Continuous lines and the degreeof curving can be used to interpret the complexity of the image.

Curves—Curves are a subset of continuous lines, the collection of whichare employed to measure the roundness of an image. To achieve this, weconsider each curve as a section of an ellipse, thus we use ellipses tofit continuous lines. Fitted curves are represented by parameters of itscorresponding ellipses.

Capturing Emotion from Shapes

For decades, numerous theories have been promoted that are focused onthe relationship between emotions and the visual characteristics ofsimplicity, complexity, roundness, and angularity. Despite thesetheories, researchers have yet to resolve how to model theserelationships quantitatively. We propose to use shape features tocapture those visual characteristics. By identifying the link betweenshape features and emotions, we are able to determine the relationshipbetween the aforementioned visual characteristics and emotions.

We now present the details of the proposed shape features: linesegments, angles, continuous lines, and curves. A total of 219 shapefeatures are summarized in Table I.

TABLE I Summary of shape features. Category Short Name # LineOrientation 60 Segments Length 11 Mass of the image 4 Continuous Degreeof curving 14 Lines Length span 9 Line count 4 Mass of continuous lines4 Angles Angle count 3 Angular metrics 35 Curves Fitness 14 Circularity17 Area 8 Orientation 14 Mass of curves 4 Top round curves 18

Line Segments

Psychologists and artists have claimed that the simplicity-complexity ofan image is determined not only by lines or curves, but also by itsoverall structure and support [3]. Based on this idea, we employed linesegments extracted from images to capture their structure. Particularly,we used the orientation, length, and mass of line segments to determinethe complexity of the images.

Orientation—To capture an overall orientation, we employed statisticalmeasures of minimum (min), maximum (max), 0.75 quantile, 0.25 quantile,the difference between 0.75 quantile and 0.25 quantile, the differencebetween max and min, sum, total number, median, mean, and standarddeviation (we will later refer to these as {statistical measures}), andentropy. We experimented with both 6- and 18-bin histograms. The uniqueorientations were measured based on the two histograms to capture thesimplicity-complexity of the image.

Among all line segments, horizontal lines and vertical lines are known[3] to be static and to represent the feelings of calm and stabilitywithin the image. Horizontal lines suggest peace and calm, whereasvertical lines indicate strength. To capture the emotions evoked bythese characteristics, we counted the number of horizontal lines andvertical lines through an 18-bin histogram.

The orientation θ, of horizontal lines fall within 0°<θ<10° or170°<θ<180°, and 80°<θ<100° for vertical lines.

Length—The length of line segments reflects the simplicity of images.Images with simple structure might use long lines to fit contours,whereas complex contours have shorter lines. We characterized the lengthdistribution by calculating the {statistical measures} of lengths ofline segments within the image.

Mass of the image—The centroid of line segments may indicate associatedrelationships among line segments within the visual design [3]. Hence,we calculate the mean and standard deviation of the x and y coordinatesof the line segments to find the mass of each image.

Some of the example images and their features are presented in FIGS. 8and 9. FIG. 8 presents the ten lowest mean values of the length of linesegments. The first row shows the original images, the second row showsthe line segments extracted from these images and the third row showsthe $18$-bin histogram for line segments in the images. The 18 binsrefer to the number of line segments with an orientation of[−90+10(i−1), −90+10i) degrees where iε{1, 2, . . . , 18}. Similarly,FIG. 9 presents the ten highest mean values of the length of linesegments.

These two figures indicate that the length or the orientation cannot beexamined separately to determine the simplicity-complexity of the image.Lower mean values of the length of line segments might refer to eithersimple images such as the first four images in FIG. 8 or highly compleximages such as the last four images in that figure. The histogram of theorientation of line segments helps us to distinguish the complex imagesfrom simple images by examining variation of values in each bin.

Angles

Angles are important elements in analyzing the simplicity-complexity andthe angularity of an image. We capture the visual characteristics fromangles through two perspectives.

-   -   Angle count—We first calculate the two quantitative features        claimed by Julian Hochberg, who has attempted to define        simplicity (he used the value-laden term “figural goodness”) via        information theory: “The smaller the amount of information        needed to define a given organization as compared to the other        alternatives, the more likely that the figure will be so        perceived” [3]. Hence this minimal information structure is        captured using the number of angles and the percentage of unique        angles in the image.    -   Angular metrics—We use the {statistical measures} to extract        angular metrics. We also calculate the 6- and 18-bin histograms        on angles and their entropies.

Some of the example images and features are presented in FIGS. 10 and11. Images with lowest and highest number of angles are shown along withtheir corresponding contours in FIG. 10. These examples show promisingrelationships between angular features and simplicity-complexity of theimage. Example results for the histogram of angles in the image arepresented in FIG. 11. The 18 bins refer to the number of line segmentswith an orientation in [10(i−1), 10i) degrees where iε{1, 2, . . . ,18}.

Continuous Lines

We attempt to capture the degree of curvature from continuous lines,which has implications for the simplicity-complexity of images. We alsocalculated the number of continuous lines, which is the thirdquantitative feature specified by Julian Hochberg [3]. For continuouslines, open/closeness are factors affecting the simplicity-complexity ofan image. In the following, we focus on the calculation of the degree ofcurving, the length span value, and the number of open lines and closedlines. The length span refers to the highest Euclidean distance amongall pairs of points on the continuous lines.

$\begin{matrix}{{{{LengthSpan}\mspace{11mu} (l)} = {\max\limits_{{p_{i} \in l},{p_{j} \in l}}\mspace{14mu} {{EuclideanDist}\mspace{11mu} \left( {p_{i},p_{j}} \right)}}},} & (1)\end{matrix}$

where {p₁,p₂,K,p_(N)} are the points on continuous line l.

-   -   Degree of curving—We calculated the degree of curving of each        line as

Degree of curving(l)=Length Span(l)/N,  (2)

where N is the number of points on continuous line l.

To capture the statistical characteristics of contiguous lines in theimage, we calculated the {statistical measures}. We also generated a5-bin histogram on the degree of curving of all continuous lines (FIGS.12 and 13).

-   -   Length span—We used {statistical measures} for the length span        of all continuous lines.    -   Line count—We counted the total number of continuous lines, the        total number of open lines, and the total number of closed lines        in the image.

Curves

We used the nature of curves to model the roundness of images. For eachcurve, we calculated the extent of fit to an ellipse as well as theparameters of the ellipse such as its area, circularity, and mass ofcurves. The curve features are explained in detail below.

-   -   Fitness, area, circularity—The fitness of an ellipse refers to        the overlap between the proposed ellipse and the curves in the        image. The area of the fitted ellipse is also calculated. The        circularity is represented by the ratio of the minor and major        axes of the ellipses. The angular orientation of the ellipse is        also measured. For each of the measures, we used the        {statistical measures} and entropies of the histograms as the        features to depict the roundness of the image.    -   Mass of curves—We used the mean value and standard deviation of        {x, y} coordinates to describe the mass of curves.    -   Top round curves—To make full use of the discovered curves and        to depict roundness, we included the fitness, area, circularity,        and mass of curves for each of the top three curves.

To examine the relationship between curves and positive-negative images,we calculated the average number of curves in terms of values ofcircularity and fitness on positive images (i.e., the value is higherthan 6 in the dimension of valance) and negative images (i.e., the valueis lower than 4.5 in the dimension of valance).

The results are shown in Tables II and III. Positive images have morecurves with 60%-100% fitness to ellipses and higher average curve count.

TABLE II Average number of curves in terms of the value of fitness inpositive and negative images. (0.8, 1] (0.6, 0.8] (0.4, 0.6] (0.2, 0.4]Positive imgs 2.12 9.33 5.7 2.68 Negative imgs 1.42 7.5 5.02 2.73

TABLE III Average number of curves in terms of the value of circularityin positive and negative images. (0.8, 1] (0.6, 0.8] (0.4, 0.6] (0.2,0.4] Positive imgs 0.96 2.56 5.1 11.2 Negative imgs 0.73 2.19 4 9.75

Experiments

To demonstrate the relationship between proposed shape features and thefelt emotions, the shape features were utilized in three tasks. First,we distinguished images with strong emotional content from emotionallyneutral images. Second, we fit valence and arousal dimensions usingregression methods. We then performed classification on discreteemotional categories. The proposed features were compared with thefeatures discussed in Machajdik et al. [18], and overall accuracy wasquantified by combining those features. Forward selection and PrincipalComponent Analysis (PCA) strategies were employed for feature selectionand to find the best combination of features.

Dataset

We used two subsets of the IAPS [15] dataset, which were developed byexamining human affective responses to color photographs with varyingdegrees of emotional content. The TAPS dataset contains 1,182 images,wherein each image is associated with an empirically derived mean andstandard deviation of valance, arousal, and dominance ratings.

Subset A of the TAPS dataset includes many images with faces and humanbodies. Facial expressions and body language strongly affect emotionsaroused by images, slight changes of which might lead to an oppositeemotion. The proposed shape features are sensitive to faces hence weremoved all images with faces and human bodies from the scope of thisstudy. In experiments, we only considered the remaining 484 images,which we labeled as Subset A}. To provide a better understanding of theratings of the dataset, we analyzed the distribution of ratings withinvalence and arousal, as shown in FIGS. 14A and 14B. We also calculatedaverage variations of ratings in each rating unit (i.e., 1-2, 2-3, . . ., 7-8). Valence ratings between 3 and 4, and 6 and 7, have the maximumvariance for single images. Similarly, arousal ratings between 4 and 5varied the most.

Subset B are images with category labels (with discrete emotions),generated by Mikels [19]. Subset B includes eight categories namely,anger, disgust, fear, sadness, amusement, awe, contentment, andexcitement, with 394 images in total. Subset B is a commonly useddataset, hence we used it to benchmark our classification accuracy withthe results mentioned in Machajdik et al. [18].

Identifying Strong Emotional Content

Images with strong emotional content have very high or very low valanceand arousal ratings. Images with values around the mean values ofvalance and arousal lack emotions and were used as samples foremotionally neutral images.

Based on dimensions of valance and arousal respectively, we generatedtwo sample sets from Subset A. In Set 1, images with valence valueshigher than 6 or lower than 3.5 were considered images with strongemotional content and the rest to represent emotionally neutral images.This resulted in 247 emotional images and 237 neutral images. Similarly,images with arousal values higher than 5.5 or lower than 3.7 weredefined as emotional images, and others as neutral images. With similarthresholds, we obtained 239 emotional images and 245 neutral images inSet 2.

We used the traditional Support Vector Machines (SVM) with radial basisfunction (RBF) kernel to perform the classification task. We trained SVMmodels using the proposed shape features, Machajdik's features, andcombined (Machajdik's and shape) features. Training and testing wereperformed by dividing the dataset uniformly into training and testingsets. As we removed all images with faces and human bodies, we did notconsider facial and skin features discussed in [18]. We used bothforward selection and PCA methods to perform feature selection. In theforward selection method, we used the greedy strategy and accumulatedone feature at a time to obtain the subset of features that maximizedthe classification accuracy. The seed features were also chosen atrandom over multiple iterations to obtain better results. Our analysesshowed that the forward selection strategy achieved greater accuracy forSet 2, whereas PCA performed better for Set 1 (FIGS. 15A-15B). Thefeature comparison showed that the combined (Machajdik's and shape)features achieved the highest classification accuracy, whereasindividually the shape features alone were much stronger than thefeatures from [18] (Machajdik's features). This result is intuitivesince emotions evoked by images cannot be well represented by shapesalone and can definitely be bolstered by other image features includingtheir color composition and texture.

By analyzing valence and arousal ratings of the correctly classifiedimages, we observed that very complex/simple, round and angular imageshad strong emotional content and high valence values. Simple structuredimages with very low degrees of curving also tends to portray strongemotional content as well as to have high arousal values. By analyzingthe individual features for classification accuracy we found that linecount, fitness, length span, degree of curving, and the number ofhorizontal lines achieved the best classification accuracy in Set 1.Fitness and line orientation were more dominant in Set 2.

We present a few example images, which were wrongly classified based onthe proposed shape features in FIGS. 16 and 17. The misclassificationcan be explained as a shortcoming of the shape features in understandingthe semantics. Some of the images generated extreme emotions based onimage content irrespective of the low-level features. Besides thesemantics, our performance was also limited by the performance of thecontour extraction algorithm.

Fitting the Dimensionality of Emotion

Emotions can be represented by word pairs, as previously done in [23].However, some emotions are difficult to label. Modeling basic emotionaldimensions helps in alleviating this problem. We represented emotion asa tuple consisting of valence and arousal values. The values of valenceand arousal were in the range of (1, 9). In order to predict the valuesof valence and arousal we proposed to learn a regression model foreither dimension separately.

We used SVM regression with RBF kernel to model the valance and arousalvalues using shape, Machajdik's features, as well as the combination offeatures. The mean squared error (MSE) was computed for each of theindividual features as well as combined for both valence and arousalvalues separately. The MSE values are shown in FIG. 18A. These figuresshow that the valance values were modeled more accurately by Machajdik'sfeatures than our shape features. Arousal was well modeled by shapefeatures with a mean squared error of 0.9. However, the combined featureperformance did not show any improvements. The results indicated thatvisual shapes provide a stronger cue in understanding the valence asopposed to the combination of color, texture, and composition in images.

We also computed the correlation between quantified individual shapefeatures and valence-arousal ratings. The higher the correlation, themore relevant the features were. Through this process we found thatangular count, fitness, circularity, and orientation of line segmentsshowed higher correlations with valance, whereas angle count, anglemetrics, straightness, length span, and orientation of curves had highercorrelations with arousal.

Classifying Categorized Emotions

To evaluate the relationship between shape features and emotions ondiscrete emotions, we classified images into one of the eightcategories, anger, disgust, fear, sadness, amusement, awe, contentment,and excitement. We followed Machajdik et al. [18] and performedone-versus-all classification to compare and benchmark ourclassification accuracy. The classification results are reported in FIG.18B. We used SVM to assign the images to one of the eight classes. Thehighest accuracy was obtained by combining Machajdik's with shapefeatures. We also observed a considerable increase in the classificationaccuracy by using the shape features alone, which proves that shapefeatures indeed capture emotions in images more effectively.

In this experiment, we also built classifiers for each of the shapefeatures. Each of the shape features listed in Table IV achieved aclassification accuracy of 30% or higher.

TABLE IV Significant features to emotions. Emotion Features AngryCircularity Disgust Length of line segments Fear Orientation of linesegments and angle count Sadness Fitness, mass of curves, circularity,and orientation of line segments Amusement Mass of curves andorientation of line segments Awe Orientation of line segments ExcitementOrientation of line segments Contentment Mass of lines, angle count, andorientation of line segments

An Investigation into Three Visual Characteristics of Complex Scenesthat Evoke Human Emotion

Prior computational studies have examined hundreds of visualcharacteristics related to color, texture, and composition in an attemptto predict human emotional responses. Beyond those features, roundness,angularity, and complexity have also been found to evoke emotions inhuman perceivers, with evidence from psychological studies of facialexpressions, dancing poses, and even simple synthetic visual patterns.Capturing these characteristics algorithmically to incorporate incomputational studies, however, has proven difficult. Here we expand thescope of previous work by examining these three visual characteristicsin computer analysis of complex scenes, and compare the results to thehundreds of visual qualities previously examined.

A large collection of ecologically valid stimuli (i.e., photographshumans regularly encounter on the web), containing more than 40K imagescrawled from web albums, were generated using crowdsourcing andsubjected to human subject emotion ratings. We developed computationalmethods to map visual content to the scales of roundness, angularity,and complexity as three new computational constructs. Critically, thesethree new visual constructs achieved comparable classification accuracyto the hundreds of shape, texture, composition, and facial featurecharacteristics previously examined. In addition, our experimentalresults showed that color features related most strongly with thepositivity of perceived emotions, the texture features related more tocalmness or excitement, and roundness, angularity, and complexityrelated consistently with both dimensions of emotions.

Approach The EmoSet

To have a large collection of photographs with complex scenes, wecrawled more than 50K images from FLICKR, one of the most popular Webalbums. We performed human subject study on those photographs anddeveloped a large-scale ecologically valid image stimuli, i.e., TheEmoSet. The human subject study was empowered by crowdsourcing andcomputational tools, where we incorporated those strict psychologicalprocedures into the User Interface (UI) design, in order to recruit adiverse population of human subjects.

As a result, the EmoSet contains 43,837 color images associated withemotional labels, including dimensional labels, categorical labels, andlikeliness ratings. Subjects' demographics were also collected such asage, gender, ethnic groups, nationality, educational background, andincome level. Besides, we have collected all the semantic tags and othermetadata associated with images in the EmoSet. We will publish theEmoSet for non-commercial research upon the publication of the paper.

The Data Collection Approach

To collect image stimuli, we took 558 emotional words summarized byAverill [29] and used those words to retrieve images by triggering theFLICKR image search engine (Examples are presented in FIG. 19). For eachemotional word, we took the top 100 returned images to ensure a highcorrelation between images and the query. The crawled images weregenerated by Web users and contained complex scenes that human mayencounter in daily life. We removed duplicate images, images of badtaste, and the ones fully-occupied by text.

The Human Subject Study

In our efforts to establish a large-scale image stimuli, we leveragedcrowdsourcing and computational tools, and collect the immediateaffective responses from human subjects given the visual stimuli. Ifsubjects needed to refer back to the image, they were allowed to click“Reshow Image” in the upper left part of the screen, and click “Hide” toreturn to the three parts.

Dataset Statistics

We statistically analyze the EmoSet, including the collected emotionallabels and subjects' demographics. Each image in the EmoSet wasevaluated by at least three subjects. To reduce low-quality ratings, weremoved ratings with viewing duration shorter than 2.5 seconds. Thescale of valence, arousal, and dominance is from 1 to 9, the same withthe one in IAPS, and the range of likeliness is from 1 to 7, the samerating scale with the widely used photo.net Website. We show thedistributions of mean values in valence, arousal, dominance, andlikeliness in FIGS. 20A-20D.

The human subject study involved both psychology students within PennState University and users on the Amazon Mechanical Turk, which ensuresa diverse population of emotional ratings. Among the 4148 human subjectswe recruited, there were 2236 females and 1912 males, with age rangingfrom 18 to 72, various ethnic groups including American Indian or AlaskaNative, Asian, African American, native Hawaiian or Other PacificIslander, Hispanic or Latino, and Not Hispanic or Latino. Those humansubjects also had various income and education levels.

Constructs of Roundness, Angularity, and Simplicity

To investigate the three visual characteristics of complex scenes thatevoke human emotion, this paper proposed computational methods to mapimages to the scales of roundness, angularity, and complexity as threenew computational constructs. We detailed the three constructs in thefollowing sections.

Roundness

Roundness was defined as “the measure of how closely the shape of anobject approached that of a circle.” [30]. To compute the roundnessscore of an image, we first segmented the image into regions, thentraced their boundaries, and finally computed the goodness of fit to acircle for each region. The step-by-step procedure is:

1) The segmentation approach in [31] was adopted. Suppose the segmentsare S={S1, S2, . . . , SN}, where the number of segments wasautomatically determined by the algorithm. Let the set of boundarypoints of segment Si be Bi={(xj, yj)}.

2) The Pratt Algorithm [32] was applied to find the circle Ci bestfitted to Bi. Denote the center of the circle by (ci, di) and radius byui. The Pratt Algorithm was applied because of its capacity to fitincomplete circles, i.e., arcs of any degree.

3) For each segment, we defined the roundness disparity of Si byri=σ(d(Bi, Ci)). Denote by d(Bi, Ci) a set of distance between eachpoint in Bi to Ci, and denote by σ the standard deviation of that set.The distance between a point (xi, yi) and a circle Ci was computed bythe absolute difference between the radius ui and the Euclidean distancefrom the point to the center of the circle.

4) The roundness disparity of an image I was denoted byrI=minNrie−λri/max(v,h). Denote by v the number rows and h the number ofcolumns of the image I.

In the experiments, we set λ=0.5, and normalized the roundness disparityvalues to [0, 1] and set the roundness score to be 1−rI. Hence thecloser rI was to 1 meant that the image was associated with an obviousround property and 0 the opposite. We present examples of images andtheir roundness scores in FIG. 21. The images with highest roundnessscores were shown in the first row; images with medium ranges ofroundness scores in the second row; and images with lowest roundnessscores in the third row.

Angularity

In the Merriam-Webster dictionary, angularity is defined as “the qualityof being angular”, and angular is explained as being lean and havingprominent bone structure. We also interviewed five subjects, includingone undergraduate student, three graduate students, and one facultymember. The faculty member remarked that angular images in his mindreferred to “sword-like” images. The college student said that tallbuildings/architectures with angular shapes reflected his perception ofangularity. The three graduate students gave examples such as streets,cubics, and tall and lean buildings. These clues motivated us to examinehow similar object boundaries are to long ellipses. Similarly as withroundness, an image was segmented into regions, for each of which anangularity measure was computed.

We approximated the quality of being lean and having prominent bonestructure by the elongatedness of fitted ellipses. Specifically, theangularity score of an image was computed as follows:

1) For each set Bi, least-squares criterion was used to estimate thebest fit to an ellipse Ei. Denote the center of the ellipse by (ci, di),semimajor axis by mi, semiminor axis by ni, and angle of the ellipse byei.

2) For each image segment Si, denote the angularity of region i byai=mi/ni. As our goal is to find lean ellipses, we omitted horizontaland vertical ellipses according to ei. So did ellipses that were toosmall.

3) We computed angularity of the image I, denote by aI=maxN ai.

Angularity scores for images in the EmoSet were computed and normalizedto [0, 1]. The closer aI was to 1 meant that the image showed an obviousangular property. Examples of images and their angularity scores arepresented in FIG. 22.

Simplicity

According to [3], simplicity (complexity) of an image is primarilydepending on two objective factors: Minimalistic structures that areused in a given representation and the simplest way of organizing thesestructures. Motivated by such concept, we used the number of segments inan image as an indication for its simplicity. We defined the simplicity(complexity) score by si=|S| and normalized the scores to [0, 1] forimages in the EmoSet. The simplicity and complexity were essentiallyrepresented by the same construct, we thus omitted complexity in thelater presentations. We present examples of images and their scores ofsimplicity (complexity) in FIG. 26.

Findings

In this section, we present the three major findings of the study, i.e.,statistical correlations between roundness, angularity, and simplicityand human emotion (Statistical Correlations), the capacity of the threeconstructs in classifying the positivity of perceived emotion (The ThreeConstructs), and the power of various visual features in classifying thepositivity and calmness of perceived emotion (Visual Characteristics).

Whereas psychological conventions treated roundness and angularity asopposite properties, some natural photographs showed neither of theproperties. As the goal of the study is to examine the capabilities ofroundness, angularity, and simplicity in evoking human emotion, wetargeted visual stimuli with at least a non-zero construct of roundnessor angularity. We thus removed 12, 158 images from the EmoSet where theyare associated with zero constructs of both roundness and angularity,which results in 31,679 images.

Statistical Correlations

To examine the intrinsic relationship between the three constructs andevoked emotion, we computed correlations between one construct, such assimplicity, roundness, and angularity, and a dimension of the emotionalresponse, such as valence, arousal, and dominance, and found all thecorrelations are statistically significant, except the correlationbetween roundness and likeliness. The results are shown in FIGS. 24, 25,and 26. The red number at the top left corner indicated thestatistically significant correlations in terms of p-value.

In particular, the strongest correlation coefficient is betweensimplicity and valence, i.e., 0.11. The correlation coefficients betweensimplicity and arousal, dominance, and likeliness are 0.09, 0.04, and0.07, receptively. Whereas the correlation coefficients are smallnumerically from a psychological perspective, they are computed on 31,679 images containing complex back-ground and evaluated in uncontrolleduser subject study settings. As the p-value is much smaller than 0.0001,the intrinsic relationships between simplicity and four dimensions ofperceived emotions were indicated. Similarly, for angularity, itscorrelation coefficients with valence, arousal, dominance, andlikeliness are 0.05, 0.03, 0.02, and 0.05, and for roundness, resultsare 0.01, 0.02, 0.02, and −0.01. The correlation coefficients betweenangularity, roundness and perceived emotion are smaller than simplicity,which implied that the simplicity relates stronger with perceivedemotion compared with angularity and roundness on an arbitraryphotograph.

TABLE V Classification Results of High Valence vs. Low Valence Featuresroundness angularity simplicity 3-constructs color color + 3-conC.T.C.F. + shape C.T.C.F. + 3-con Dimension 1 1 1 3 70 73 332 116Accuracy(%) 51.08 54.75 57.36 58.08 64.42 64.97 64.86 65.5

TABLE VI Emotion Classification Results Features 3-constructs colortexture composition shape Dimension 3 70 26 13 219 Accuracy 58.08 64.4261.47 62.58 60.19 (valence)(%) Accuracy 56.1 58 59.55 56.15 58.7(arousal)(%)

The Three Constructs

To examine the capacity of the three constructs for classifying theemotional responses of natural image stimuli, we formulated aclassification task to distinguish positive emotions from negative ones,i.e., high and low valence. The scale of valence is from 1 to 9, where 1refers to the lowest value in valence and 9 the highest. Images with amedium-range score, such as 5, show neither positive emotions nornegative emotions. Following conventions in computer science that a gapmay apply to facilitate classifier training, we adopted a gap of 1.87 todivide image collections into two groups, images arousing positiveemotions (valence >6.63) and negative emotions (valence <4.5). To adjustthe classifier parameters and evaluate the trained classifier, werandomly divided the data into training, validation, and testing sets,where the number of images with positive and negative emotions wasequal. Specifically, we randomly selected 70% of the data used fortraining, 10% for validation, and 20% for testing. This resulted in12600 images in training, 1800 images for validation, and 3600 imagesfor testing. The SVM classifier with RBM kernel was applied, as it wasone of the best classifier training approaches in computer science.Among the 144 pairs of parameter candidates, the best-performed c and gwere selected given their performance on the validation dataset.

Various visual features were used in the classification. Color, texture,facial, and composition features were computed as presented in [18] andshape features were as in Section 4. The three constructs were computedas described in the section on approaches. “3-constructs” refers to theconcatenation of the three constructs, and “color+3-con” denotes theconcatenation of color features and the three constructs.“C.T.C.F.+3-con” refers to a con-catenation of color, texture,composition, facial, and the three constructs. “C.T.C.F.+shape” refersto the concatenation of color, texture, composition, facial, and shapefeature. Results are presented in Table 5. As shown in the Table, the 73dimensional “color+3-con” feature improved upon the “color” featureslightly. Compared to the 332 dimensional features, “color+3-con”achieved a competitive and even better classification results usinglow-dimensional features (73 dimensions). We also noticed that the bestclassification results were achieved by “C.T.C.F.+3-con”, which clearlyshowed the capabilities of roundness, angularity, and simplicity ofevoking the positivity or negativity of human emotion.

Visual Characteristics

To examine the visual characteristics of complex scenes of evokingdifferent dimensions of emotion, we classified the calmness of emotions,i.e., high and low arousal following a similar setting with thepositivity of emotions. First, a gap was adopted of 2.7 to divide imagecollections into two groups, high arousal (arousal >6.4) and low arousal(arousal <3.7). Then, training, testing, and validation sets weregenerated, where 7000 images were used in training, 1000 images forvalidation, and 2000 images for testing. Finally, the SVM classifier wastrained and the best set of parameters was selected according to theirperformance on the validation set. We compared classification results invalence and arousal using color, texture, composition, shape, and thethree constructs. The results were presented in Table 6. As shown in theTables 5 and 6, color features performed the best among the five featuregroups for distinguishing high-valence images from low-valence ones.Texture and shape features performed better at classifying images thataroused calm emotions and excited emotions. The three constructs showedconsistent predictability for both of the classification tasks usingmerely the three numbers as the predictors.

Targeted Applications of Automatic Emotion Prediction

All of the methods and steps disclosed herein are implemented on aprogrammed digital computer, which may be a stand-alone machine orintegrated into another piece of equipment such as a digital still orvideo camera including, in all embodiments, portable devices such assmart phones. The invention thus finds wide application in many fieldsof use, including the ability to improve technologies outside digitalimage interpretation or classification. As one example, the computerizedsystem provides a photographer or a videographer with assessments on theemotions that the viewers can potentially experience while viewing thephotos. In this case, the digital images may be acquired by a camera andtransferred to a computer over a wired or wireless connection or networkfor automatic emotion analysis or, alternatively, the computer may beembedded in a still or video camera enabling the photographer orvideographer to assess emotional content before or after storage in amemory device in the camera. The embedded computer may also receivenon-visual information about the image. The non-visual information mayinclude camera setting parameters and textual information about theimage. The emotional content of the image may be obtained based upon themodeled shape features and one or both of the extracted visual featuresand non-visual information.

In other applications, the computer system evaluates images or videos topredict whether the viewers will experience positive or negativeemotions, and how strong those emotions are. Such images may beadvertising images, political campaign images, movies or movie trailers,and so forth. A movie director may use the computer system to estimatethe emotional response of the audience for a video clip; the computersystem can assist political campaign designers develop visual elementssuch as images or videos that can evoke certain feelings in viewers.Product designers can use the computer system to estimate the emotionalresponse of shoppers based on the visual appearance of the product. Incases such as these, the invention extends, improves upon and/orenhances fields of endeavor outside digital image interpretation,including advertising, marketing, sales, politics and fundraising.

The invention may be coupled to a search engine, thereby providingemotion assessment to rank visual search results so that images orvideos of certain emotions can be presented to the users. In thisinstance, the search engine, running on a computer or portableelectronic device such as a smart phone, may include an input from auser indicating the type or strength of emotional content associatedwith a desired search, with the results reflecting that filter. Ateacher of young kids can illustrate architectural or naturalattractions using the most beautiful pictures retrieved by a searchengine capable of ranking aesthetics.

In social media, the computer system can help users select photos orvideos that evoke a desired emotion to share with other users.Particularly when integrated into a still/video camera, including smartphone embodiments, the system may provide photo or video editingsoftware operative to suggest photo or video variants (e.g., cropping,color variations, filtering) to evoke certain desired emotions on theviewers. Such suggestions may be provided before or after image capture;for example, the user of a camera may be allowed to pan/tilt/zoom whilewatching a display indicating emotional content, thereby enabling theuser to begin recording if/when a desired emotional response isachieved.

The invention may also be used to modify and/or improve upon applicationareas related to sociology, psychiatry and the like. As one example, aclinical psychologist can ask the computer to pick out a sequence ofphotographs or videos with a certain emotional characteristics for usein a therapeutic session. In the fields of robotics, machine vision andvehicular control, a computer programmed in accordance with theinvention may be used to estimate a human user's emotional state basedon the visual scene the human is experiencing, and provide correspondingcommunication such as a dialogue with the human user. A route planningor navigation program can provide users with a route that would evokecertain emotions. For example, a passenger of vehicle, including anautonomous vehicle, may request a route that provides high positiveemotions or particular kinds of emotions such as contentment orpleasure.

Interior designers, architects, stage designers and builders may enlistthe help of the computer when trying to pick out groups of photographsfor proposed constructions or decorations so as to improve the moodsexposed individuals (e.g., patients in a hospital, workers in an officebuilding, city visitors, theatregoers).

CONCLUSIONS

We investigated the computability of emotion through shape modeling. Toachieve this goal, we first extracted contours from complex images, andthen represented contours using lines and curves extracted from images.As we discussed above, we extracted locally optimized line segments byconnecting neighboring pixels from the contours extracted from the imageand formed lines and curves using the locally optimized line segments.The formed lines and curves are locally meaningful. Statistical analyseswere conducted on locally meaningful lines and curves to represent theconcept of roundness, angularity, and simplicity, which have beenpostulated as playing a key role in evoked emotion for years. Leveragingthe computational representation of these physical stimulus properties,we evaluated the proposed shape features through three tasks:distinguishing emotional images from neutral images; classifying imagesaccording to categorized emotions; and fitting the dimensionality ofemotion based on proposed shape features. We have achieved animprovement over the state-of-the-art solution [18]. We also attackedthe problem of modeling the presence or absence of strong emotionalcontent in images, which has long been overlooked. Separating imageswith strong emotional content from emotionally neutral ones can aid inmany applications including improving the performance of keyword basedimage retrieval systems. We empirically verified that our proposed shapefeatures indeed captured emotions in the images. The area ofunderstanding emotions in images is still in its infancy and modelingemotions using low-level features is the first step toward solving thisproblem. We believe our contribution takes us closer to understandingemotions in images. In the future, we hope to expand our experimentaldataset and provide stronger evidence of established relationshipsbetween shape features and emotions.

Building upon the shape features, we have also investigated three visualcharacteristics of complex scenes that evoked human emotion utilizing alarge collection of ecologically valid image stimuli. Three newconstructs were developed that mapped the visual content to the scalesof roundness, angularity, and simplicity. Results of correlationalanalyses, between each construct and each dimension of emotionalresponses, showed that some of the correlations are statisticallysignificant, e.g., simplicity and valence, angularity and valence. Andclassification results demonstrated the capacity of the three constructsin classifying both dimensions of emotion. Interestingly, by combiningwith color features, the three constructs showed comparableclassification accuracy on distinguishing positive emotions fromnegative ones as a set of 200 texture, composition, facial, and shapefeatures. As future work, the proposed approach could be easily appliedto examine other visual characteristics that evoke human emotion incomplex scenes. We expect that our efforts may contribute to researchregarding visual characteristics of complex scenes and human emotionfrom perspectives of visual arts, psychology, and computer science.

REFERENCES

-   [1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour    detection and hierarchical image segmentation. IEEE TPAMI,    33(5):898-916, 2011.-   [2] S. Arifin and P. Y. K. Cheung. A computation method for video    segmentation utilizing the pleasure-arousal-dominance emotional    information. In ACM MM, pages 68-77, 2007.-   [3] R. Arnheim. Art and visual perception: A psychology of the    creative eye. 1974.-   [4] J. Aronoff. How we recognize angry and happy emotion in people,    places, and things. Cross-Cultural Research, 40(1):83-105, 2006.-   [5] M. Bar and M. Neta. Humans prefer curved visual objects.    Psychological Science, 17(8):645-648, 2006.-   [6] M. M. Bradley and P. J. Lang. The international affective    picture system (IAPS) in the study of emotion and attention. In    Handbook of Emotion Elicitation and Assessment, pages 29-46, 2007.-   [7] S. Brandt, J. Laaksonen, and E. Oja. Statistical shape features    in content-based image retrieval. In ICPR, pages 1062-1065, 2000.-   [8] A. Chia, D. Rajan, M. Leung, and S. Rahardja. Object recognition    by discriminative combinations of line segments, ellipses and    appearance features. IEEE TPAMI, 34(9):1758-1772, 2011.-   [9] G. Csurka, S. Skaff, L. Marchesotti, and C. Saunders. Building    look & feel concept models from color combinations. The Visual    Computer, 27(12):1039-1053, 2011.-   [10] R. Datta, D. Joshi, J. Li, and J. Z. Wang. Studying aesthetics    in photographic images using a computational approach. In ECCV,    pages 288-301, 2006.-   [11] R. Datta, J. Li, and J. Z. Wang. Algorithmic inferencing of    aesthetics and emotion in natural image: An exposition. In ICIP,    pages 105-108, 2008.-   [12] A. Hanjalic and L. Q. Xu. Affective video content    representation and modeling. IEEE Trans. On Multimedia,    7(1):143-154, 2005.-   [13] D. Joshi, R. Datta, E. Fedorovskaya, Q. T. Luong, J. Z.    Wang, J. Li, and J. Luo. Aesthetics and emotions in images. IEEE    Signal Processing Magazine, 28(5):94-115, 2011.-   [14] P. J. Lang, M. M. Bradley, and B. N. Cuthbert. Emotion,    motivation, and anxiety: Brain mechanisms and psychophysiology.    Biological Psychiatry, 44(12):1248-1263, 1998.-   [15] P. J. Lang, M. M. Bradley, and B. N. Cuthbert. International    affective picture system: Affective ratings of pictures and    instruction manual. In Technical Report A-8, University of Florida,    Gainesville, Fla., 2008.-   [16] M. K. Leung and Y.-H. Yang. Dynamic two-strip algorithm in    curve fitting. Pattern Recognition, 23(1-2):69-79, 1990.-   [17] K. A. Lindquist, T. D. Wager, H. Kober, E. Bliss-Moreau,    and L. F. Barrett. The brain basis of emotion: A meta-analytic    review. Behavioral and Brain Sciences, 173(4):1-86, 2011.-   [18] J. Machajdik and A. Hanbury. Affective image classification    using features inspired by psychology and art theory. In ACM MM,    pages 83-92, 2010.-   [19] J. Mikel, B. L. Fredrickson, G. R. Larkin, C. M.    Lindberg, S. J. Maglio, and P. A. Reuter-Lorenz. Emotional category    data on images from the international affective picture system.    Behavior Research Methods, 37(4):626-630, 2005.-   [20] Y. Mingqiang, K. Kidiyo, and R. Joseph. A survey of shape    feature extraction techniques. Pattern Recognition, pages 43-90,    2008.-   [21] R. Reber, N. Schwarz, and P. Winkielman. Processing fluency and    aesthetic pleasure: Is beauty in the perceiver's processing    experience? Personality and Social Psychology Review, 8(4):364-382,    2004.-   [22] H. R. Schiffman. Sense and Perception: An Integrated Approach.    1990.-   [23] T. Shibata and T. Kato. Kansei image retrieval system for    street landscape-discrimination and graphical parameters based on    correlation of two image systems. In International Conference on    Systems, Man, and Cybernetics, pages 274-252, 2006.-   [24] M. Solli and R. Lenz. Color based bags-of-emotions. LNCS,    5702:573-580, 2009.-   [25] H. L. Wang and L. F. Cheong. Affective understanding in film.    IEEE TCSVT, 16(6):689-704, 2006.-   [26] V. Yanulevskaya, J. C. Van Gernert, K. Roth, A. K. Herbold, N.    Sebe, and J. M. Geusebroek. Emotional valence categorization using    holistic image features. In ICIP, pages 101-104, 2008.-   [27] H. Zhang, E. Augilius, T. Honkela, J. Laaksonen, H. Gamper,    and H. Alene. Analyzing emotional semantics of abstract art using    low-level image features. In Advances in Intelligent Data Analysis,    pages 413-423, 2011.-   [28] S. L. Zhang, Q. Tian, Q. M. Huang, W. Gao, and S. P. Li.    Utilizing affective analysis for efficient movie browsing. In ICIP,    pages 1853-1856, 2009.-   [29] J. R. Averill, “A semantic atlas of emotional concepts,” JSAS:    Catalog of Selected Documents in Psychology, vol. 5, no. 330, pp.    1-64, 1975.-   [30] M. Mills, Introduction to the measurement of roundness,    Taylor-Hobson Precision-   [31] J. Li, “Agglomerative connectivity constrained clustering for    im-age segmentation,” Statistical Analysis and Data Mining, vol. 4,    no. 1, pp. 84-99, 2011.-   [32] V. Pratt, “Direct least-squares fitting of algebraic surfaces,”    SIG-GRAPH Comput. Graph., vol. 21, no. 4, pp. 145-152, 1987.

1. A system for analyzing the emotional content of an image, comprisingthe steps of: a programmed computer having an input for receiving atleast one digital image, wherein the computer is programmed to performthe following functions automatically: (a) extract visual informationfrom the image; (b) model shape features in the image, and (c) computethe emotional content of the image based upon the extracted visualfeatures and modeled shape features; and an output device incommunication with the programmed computer enabling a user to receivethe computed emotional content of the image.
 2. The system of claim 1,wherein: the programmed computer also receives non-visual informationabout the image; and the computer is further operative to automaticallyoutput information regarding the emotional content of the image basedupon the modeled shape features and one or both of the extracted visualfeatures and non-visual information.
 3. The system of claim 2, wherein:the programmed computer forms part of a camera; and the non-visualinformation includes camera setting parameters.
 4. The method of claim2, further including: an input for receiving textual information aboutthe image; and wherein the non-visual information includes the textualinformation about the image.
 5. The system of claim 1, wherein the inputis configured to receive a still digital image or a plurality ofsequential digital images representing frames in a video or movie. 6.The system of claim 1, wherein the computer is further programmed toautomatically perform the following functions: use the modeled shapefeatures to determine valence or arousal coordinates based upon adimensional space model of emotions with valence and arousalcoordinates; and compute the emotional content of the image based uponthe valence or arousal coordinates of the image in the space model. 7.The system of claim 1, wherein the computer is further programmed toautomatically use the shape features to distinguished images with strongemotional content from emotionally neutral images.
 8. The system ofclaim 1, wherein the computer is further programmed to automaticallyperform the following functions: model the visual properties ofroundness, angularity and simplicity based upon on one or more shapesextracted from the image; and compute the emotional content based uponthe roundness, angularity or simplicity.
 9. The system of claim 1,wherein the computer is further programmed to automatically perform thefollowing functions: identify contours in the image; represent thecontours as lines and curves; perform statistical analyses on the linesand curves to model the visual properties of roundness, angularity, andsimplicity; and compute the emotional content based upon the roundness,angularity and simplicity.
 10. The system of claim 9, wherein thecomputer is programmed to determine if the lines or curves are locallymeaningful.
 11. The system of claim 9, wherein the computer isprogrammed to determine if the contours are strong or continuous.